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ABSTRACT 


Assessing Opportunities to Learn (OTL) implies the mea- 
surement of different aspects of the curricular implementa- 
tion in the classrooms, such as the contents that the teacher 
selects for the course and the time of exposure and the fre- 
quency of the tasks proposed. 


Based on recent studies that demonstrate the influence of 
this curricular dimension on students’ performance, this work 
analyzes the relationships between the emphasis made by 
teachers of third grade of secondary school on different read- 
ing contents, and students’ performance in reading tests. 
The aim of this research is to establish the effect of the em- 
phasis with which teachers propose different types of read- 
ing activities (literal, inferential, and critical) on students’ 
performance. From the analysis of compositional data, this 
study concludes that the students of those teachers who re- 
port working a greater extent on the critical dimension of 
reading obtains —on average— higher scores on the national 
reading test. This result holds even when socio-economic 
and cultural context and reading habits are controlled for. 
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The opportunities to learn (OTL) that the education sys- 
tems offer to students are currently one of the central ob- 
jects of educational evaluations. This implies, among other 
things, the measurement of various aspects of the curricu- 
lar implementation in the classrooms, such as: a)the con- 
tents that the teacher selects to address in the course and 
b)their exposure time and frequency according to the tasks 
proposed. As such, OTL studies have the potential to con- 
tribute to the knowledge of what and in which way the cur- 
riculum is implemented in the classroom, and to curricular 
formulations. 


One of the first organizations to use the concept of OTL in 
their studies was the International Association for the Eval- 
uation of Educational Achievement (IEA) in the Second In- 
ternational Study in Mathematics (SIMS), showing the cor- 
relation between student performance and the opportunity 
to have tackled a certain curricular content in class, see [30]. 
Likewise, the research produced from the SIMS data showed 
—among other aspects— the centrality of curricular coverage, 
emphasis, and time devoted to its treatment in relation to 
students results in tests, see [7]. 


The OTL project that served as a source for the design of 
the instruments used in the International Study of Trends 
in Mathematics and Science (TIMSS), was carried out by 
the Mathematics and Science Opportunities Survey (SMSO) 
(see [32]), which formulates the concepts of prescribed cur- 
riculum (national goals or curricula), implemented curricu- 
lum and achieved curriculum (what students have actually 
learned, see [33]. In this sense, the evaluation of opportu- 
nities to learn proposes relevant information about the im- 
plemented curriculum that contributes to a better and more 
complete understanding of what students can learn. 


Studies carried out by Cervini conclude that OTL affect stu- 
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dent performance even when controlling for variables such 
as socioeconomic status and socio-cultural context of the 
school: “students with the same social background who at- 
tend schools with a similar social composition, but some 
[have] a significantly higher performance level because their 
teachers gave them a greater opportunity to learn”, see [25]. 


Recently, and from the accumulation of evidence, greater im- 
portance has been given to the incidence of learning oppor- 
tunities in student performances. Numerous studies realize 
the importance of considering these variables when identify- 
ing the aspects that influence student performance and that 
are also modifiable from the organization of schools and the 
pedagogical practices of the classroom, see [13]. 


Consequently, when evaluating the performance of students 
based on tests aligned to the prescribed curriculum (the reg- 
ulations established to be taught in a school year) it is ex- 
pected that different results will be observed for those who 
have had greater coverage of the contents or greater empha- 
sis on its treatment, in addition to more exposure time to 
them (emphasis), than for those who have not had it, see 
[25]. The work that is developed in this article aims to give 
statistical evidence to this last hypothesis raised by Cervini 
in the Uruguayan classrooms. 


On a large scale, there are few international experiences that 
evaluate learning opportunities in the classroom: the eval- 
uations carried out by the IEA (SMSO and TIMSS), the 
International Study of Progress in Reading Comprehension 
(PIRLS), the Program for the International Assessment of 
Students (PISA) and the International Survey on Teaching 
and Learning (TALIS). Since 2017, the National Institute 
of Educational Evaluation (INEEd) of Uruguay, performs 
the national evaluation of achievements of the educational 
system (ARISTAS) that contemplates the measurement of 
OTL in classrooms of third and sixth grade of primary edu- 
cation and third grade of secondary education. 


Research based on PISA data (see [17]) has shown, for a 
large group of countries, a significant association (of mod- 
erate to strong) between reading habits (students’ taste for 
reading, time dedicated to it in their free time and the di- 
versity of texts they read) and their reading performance. 


Similar findings were reported for Uruguay by ANEP, 2004. 
In this study, the relationship between certain attitudinal 
variables regarding students’ reading habits and the results 
of reading and math tests is analyzed. In concordance with 
international findings, this study shows better reading re- 
sults for women than men. Nonetheless , it is also women 
who show greater taste for reading. Among the findings of 
this study, it is noted that as performance in reading im- 
proves, so does the taste for reading, while the dislike of it 
decreases on average. 


Although there is evidence that links student reading habits 
with their reading performance. This can happen since stu- 
dent reading habits allow some teacher practices, that in 
turn influence student performance. However, there is little 
evidence that links these qualities of students with teacher 
practices (see [4]) and their performance. Thus, this re- 
search focuses on the relative effect of teaching practices on 


the reading performance of students, when certain student 
qualities (such as socioeconomic and cultural level; and stu- 
dent reading habits) are controlled for. 


Based on data from the 2018 national evaluation of the 
Uruguayan educational system —Aristas— carried out by the 
National Institute of Educational Evaluation (INEEd), the 
aim of this work (considering recent studies that demon- 
strate the influence of curricular implementation on student 
performance) is to analyze the relationships between the em- 
phasis made by nine grade teachers on different reading di- 
mensions (literal, inferential and critical) and their student 
performance on reading. 


One possible explanation is that the students whose teachers 
report working the critical dimension of reading on a greater 
extent, achieve —on average— higher scores in the reading 
test, even when socio-economic and cultural context; and 
reading habits, are controlled for. 


This paper is organized as follows. In Section 2 the theoret- 
ical framework and the main linguistic concepts used are in- 
troduced. Section 3 provides the implemented methodology 
and the statistical tools used: Compositional Data Analysis 
(CoDA). The most relevant results obtained from the sta- 
tistical analysis are enumerated in Section 4. In Section 5 
the final discussion and conclusions are established. 


2. THEORY 


As Zakaryan (see [35]) indicates, in general terms, the na- 
ture of classroom teaching significantly affects the level of 
student learning. In this relationship, the analysis of the 
teaching practice is key, since the teacher determines the 
learning opportunities through the activities he proposes in 
the classroom and the qualities of his or her teaching. Thus, 
the content, format and cognitive demand of the tasks that 
teachers pose in the classroom constitute “the main vehicle 
to provide school children with learning opportunities”, see 
[28], p. 113. 


One of the questions which OTL studies attempt to respond 
is whether the curriculum implemented by teachers in the 
classrooms effectively covers the contents established in the 
nationally prescribed curriculum. Within this research per- 
spective, three approaches are identified: the coverage of the 
contents, the exposure to them —measured through the time 
dedicated to them— and the emphasis on their implementa- 
tion, that is, what contents are treated with priority in the 
classrooms over others in the program, see [26]. 


In this way, it is possible to approach the institutional and 
pedagogical mechanisms that contribute to the distribution 
of learning opportunities not only among schools, but also 
between classrooms. Likewise, it is relevant to know the 
form of said distribution, that is, if all children and young 
people receive from the school system —or not-— the same 
opportunities to learn. 


This work follows the definition of opportunities to learn 
(OTL) carried out by the INEEd, see [15]. In this, the study 
of the OTL not only seeks to analyze the alignment between 
the prescribed and the implemented curriculum, but also to 
what extent this alignment and other school conditions in- 
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fluence the performance of the students. Among the dimen- 
sions of the OTL assessed by the INEEd, this paper focuses 
on coverage and sequence of the contents, which refers to the 
degree of implementation of the curricular contents, as well 
as their didactic sequence and emphasis, see [15], p. 14. In 
particular, the emphasis on curricular content is measure as 
the percentage of classroom pedagogical time that teachers 
devote to the work of curricular content corresponding to a 
school grade, see [15], p. 15. 


3. METHOD 


This study uses data from the evaluation of the uruguayan 
education system (Aristas) carried out by the INEEd in 
secondary education in 2018 nationwide. This evaluation 
gathers information on various actors of the education sys- 
tem: school principals, teachers and students, through self- 
administered online surveys. Aristas is applied to a repre- 
sentative sample of nine grade middle school students of the 
Uruguayan urban region. Besides this contextual question- 
naires, each student has to take two multiple choice tests to 
assess reading and math performance (Students’ abilities are 
estimated based on test results using item response theory, 
and in particular the Rash model). 


To obtain information on curricular implementation, Aris- 
tas asks teachers if they have covered and with how much 
emphasis, a series of curricular activities that can be clas- 
sified into three reading dimensions: literal, inferential and 
critical. 


As defined by INEEd in the OTL framework for secondary 
education (see [14] and [15]), the emphasis with which teach- 
ers work the curricular contents in the classroom is a com- 
posite measure based on their reports on: i) the total number 
of classes taught and ii) the number of classes dedicated to 
addressing each type of activity in the course. Both mea- 
sures refer to the period between the beginning of the school 
year (March 2018) and the time the evaluation was carried 
out (October 2018). 


In order to address this, teachers are presented with a list of 
10 activities (corresponding to the three different dimensions 
of reading —literal, inferential, critical—-) and are asked to 
indicate the number of classes dedicated to each activity. 
Each of the dimensions in which the activities are grouped 
implies differences in their cognitive complexity, see Table 
1. Thus, the number of classes reported per dimension is 
counted as the average number of classes declared for each of 
the listed activities. That is, the number of classes reported 
by a teacher in one dimension is the sum of classes that the 
teacher declares to dictate for the activities that integrate 
that dimension, over the total of activities that compose it. 


From the self-report declared by nine grade teachers in sec- 
ondary education, an overestimation of the effective number 
of classes in the school year is generally observed. That is, 
when the maximum total number of classes in the year is 
72%, 70.2% report dictating above that threshold. This in- 
convenience in the instrument is a frequent problem due to 
various causes, see [10] and [4]. 


To approach this over report problem, the classes reported 
are relativized by each teacher for the activities of each di- 


Literal dimension 


Inferential dimension 


Critical dimension 


Recognize basic elements of 
the enunciation situation 


Recognize the subject of the 
paragraph or statement 


Evaluate and / or interpret 
the facts, situations or con- 
cepts posed by the text 


Locate explicit information 


Summarize the general idea 
of the text and draw conclu- 
sions 


Recognize items complexes 
of the enunciation situation 
(assumptions, implications, 
reasons, ideological position 
of the enunciator, intertex- 
tuality, parody, irony, exag- 
geration) 


Recognize the thematic pro- 
gression 


Recognize narrative, de- 
scriptive, argumentative or 
expository intentionality 


Kinds of activity 


Match information of sen- 
tences and paragraphs 
Hierarchize data or events 
and establish relationships 
between texts when it has 
different formats (eg tables 
and text) 


Table 1: Activities classified according to the dimension they 
conform. Source: own elaboration. 


mension of reading to the total of classes according to the 
school calendar. It is assumed that the relative weight that 
the teacher reports in each activity is a valid measure of the 
real time spent in class. 


The classes reported by the i-th teacher to the literal, infer- 
ential and critical reading respectively are called L;, [; and 
C;, so the normalized vector is 


X; = (LR, 1R;,CR;) = : 


ns a Lili, Ci . 
i Pas TAI ) 


Let S? be the unit simplex of R*® (S? = {(x,y,z) € R°/a+ 
yt+z2= 1,2 > 0,y > 0,z > OF). It is true that X; € 
S? Yi = 1,...,N, where N is the total number of teach- 
ers in the sample (N = 364). In this way, conclusions 
can be drawn about the behavior in relative but not ab- 
solute terms. The isometric transformation is then given by 


op: Rio — S?, where 4(2,y, 2) = ( 
Rio = {(x, y, z) € R?/z, y, z = O}). 


x y 2 
otyte’? otytz? wryte) ¥ 


This means that the class proportions dictated (on average) 
in each of the three dimensions of reading (literal, inferential 
and critical) are thought of as compositional data. In this 
sense, the vector is made up of three components, each of 
which represents the proportion of classes dictated in that 
dimension, therefore the components of the vector of propor- 
tions are non-negative and add to 1. This vector indicates 
the relative emphasis the teacher reports assigning to each 
of the dimensions. This type of compositional data analysis 
doesn’t allow for classical or hierarchical regression models 
as its assumptions don’t hold. 


The analysis of compositional data dates back to Pearson 
1987, but the basis of statistical theory for this type of data 
has been developing since the middle of the last century, 
see [12], [8] and [9]. The association that exists between 
the components of the vector determines a series of method- 
ological difficulties (see [34]) that lead to the need for specific 
techniques for this type of data. 


A large number of applications of compositional data theory 
are found in the literature for different fields (see for example 
[5], [23], [20] and [27]). However, there are few applications 
in reference to educational data and in particular in the area 
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of educational evaluation, see for example [31], [19], [21] and 


(6). 


On the other hand, reading achievement is assessed through 
a standardized reading test developed by INEEd. The test 
is designed based on a general statement of reading (reading 
competence) which is successively broken down into state- 
ments (dimensions of said competence: literal, inferential 
and critical) and subaffirmations (knowledge and skills). Test 
items are designed to assess students mastery of the subaffir- 
mations. From a psychometric perspective, the Item Theory 
Response (TRI) method is used for the calibration and con- 
struction of the items. For the porpoise of this study, the 
group performance or ability (Y;) is the average of the scores 
assigned by TRI to all the students of the ith—-group. 


4. RESULTS 


Data from Aristas 2018 in Uruguay shows a relationship be- 
tween socioeconomic level and reading habits of the group, 
and the emphasis the teacher assigns to certain reading ac- 
tivities over others, see [4]. Higher socioeconomic levels and 
more frequent reading habits of the group are associated 
with more teacher emphasis on the dimension of critical 
reading. Even when socioeconomic level is controlled for, 
the reading habits of the group have a significant effect over 
teacher emphasis. 


Figure 1 shows, by means of box plot, the strong relationship 
between the quintiles of the socioeconomic level of the group 
(ESCS) and the average reading performance obtained by 
the group of students (left panel). Moreover, higher reading 
habits of the group also have an effect, (not as pronounced 
as ESCS) on student performance (right panel). 
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Figure 1: Left Panel: Box plot of the average performance of 
the group in the reading test according to the socioeconomic 
quintile of the group. Right Panel: Box plot of the average 
performance of the group in the reading test according to 
the level of reading habits of the group. 


The relationship between reading test results and empha- 
sis placed by teachers on each of the three dimensions of 
reading, is evaluated. As Figure 2 shows, groups who per- 
form better on the test devote more time to activities that 
involve critical reading. Moreover, as group performance im- 
prove, the proportion of classes dedicated to literal reading 
decreases. These trends don’t appear to apply to inferential 
reading. 


However, these results are a) only descriptive of the sam- 
ple and b) they are marginal, that is, the joint effect of the 
activity vector on student performance and its relative im- 
portance vis-a-vis other variables of the study (ESCS and 


ESCS_Grupo_cat 


uiT INF CRIT 
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Proportion reported by teachers 


a a2 a3 a4 at a2 a3 a4 a a2 a3 a4 
Quartiles of the group performance variable 


Figure 2: Box diagrams of the proportion in each of the ac- 
tivities (literal-inferential-critical) according to the quartiles 
of the group’s performance variable. 


reading habits) is not observed. 


To model the problem, a non-parametric regression model 
(kernel method, see [18]) and a parametric regression model 
(Dirichlet regression, see [29] and [24]) are considered, see 
Figure 3. This figure reflects the average percentages of em- 
phasis placed on each dimension as a function of the group’s 
performance. On average, according to the non-parametric 
model, the emphasis on literal reading is among 20% and 
35%, decreasing as Y increases. The influence of Y on the 
proportion of activities in critical reading varies between an 
average range of 27% to 50% increasing as Y increases. The 
inferential reading is the one with the least variation as a 
function of Y, between 30% and 38%. 


In both cases it is observed that increasing group’s per- 
formance is associated to an increase in the proportion of 
classes in which critical reading activities are taught. 


INF 
100 


LIT & cS & & © CRIT 


Figure 3: Non-parametric regression by kernel method 
(points) and Dirichlet regression (continuous line) in S? 
where the independent variable is the performance of the 
group (Y). 


This result holds for the Dirichlet model. In this case the 
decrease of literal reading emphasis is associated with higher 
test results but with a moderate statistical significance (p — 
value = 0.07), see Table 2. 
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Coef. ANOVA 
Lit. Coef.. Error p-value 
Int. 0.86 0.06 < 2e-16 
Y -0.14 0.08 0.07 
Inf. Coef.. Error p-value 
Int. 0.97 0.06 < 2e-16 
Y 0 0.07 0.99 
Crit. | Coef.. Error p-value 
Int. 1.12 0.06 < 2e-16 
Y 0.46 0.07 8.94e-11 


Table 2: ANOVA of the Dirichlet regression where the in- 
dependent variable is the performance of the group (Y). 


A possible mistake when interpreting these results is to infer 
a direct association between emphasis and student perfor- 
mance when, strictly speaking, it could be an indirect effect 
caused by student reading habits and socio-economic and 
cultural level of the group, see [4]. Figure 4 shows these 
interactions. 


Group Context 


GD | - 
— > (Performance 
Cescs > ; 
Je 


Figure 4: Diagram on the relationship between activities 
and performance controlling for ESCS context variables and 
study habits. 


Literal 
Inferential 
Critical 


Activities 


To control the effect of socioeconomic status and reading 
habits on performance, two step regression is estimated. The 
performance of the group in reading Y can be decomposed 
as the sum of a 8 (hypothetical or latent) score generated by 
certain context variables (in this model they are the ESC'S 
and reading habits noted as H) and another a associated 
with the emphasis of classes in each activity, that is 


Y = Intercepto+$8+a-+e, where € is a random error centered 
and independent of a and @. In this case, student perfor- 
mance is considered as the dependent variable (as opposed 
as in Figure 3). 


The strong relationship between 6 and Y has already been 
widely studied, see [14]. If the last term a is also signif- 
icant to explain performance, this indicates that emphasis 
on a certain type of activities also influences performance, 
even controlling for certain group context variables. To “ex- 
tract” the effect of the context, the following procedure is 
performed. As a first step, to avoid the ipstative effect (see 
[11]) between the components of the vector in S? an isomet- 
ric transformation is carried out that goes from S? to R?. 
This is called the log-ratio isometric transformation (ir), 
see [1]. The image of (LR,IR,CR) is noted by (tlrz,ilry) . 


The first adjustment is then a linear model of the form, 


Y =ao+ ailrz + aiilry + bi ESCS + b2H +e (Model 1). 


Table 3 shows how all the variables are significant in this 
model, with the variables of S* tolerating the inclusion of H 
and ESCS in the model. 


Coef. ANOVA 
Model I | Coef. Error p-value 
Intercept | -0.11 0.025 4.06e-06 
ilre 0.15 0.050 0.01820 
ilry 0.12 0.042 0.00586 
ESCS 0.68 0.04 < 2e-16 
val 0.32 0.082 0.00013 
Model2 | Coef. Error r p-value 
Intercept | -0.10 0.026 < 3.44e-05 
ilre 0.15 0.053 0.00465 
ilry 0.14 0.044 0.00146 


Table 3: ANOVA considering ESC'S and H as control vari- 
ables. 


Second, after estimating the coefficients of the equation for 
maximum likelihood, the effect on the performance of the 
context variables is extracted, that is, Y,* = Yi-b1 ESCS;+ 
bo Habitos;. The final model (model 2) that allows us to 
deduce the effect of emphasis on each dimension is, 


Y* =do4+ diilrz + deilrz + €* (Model 2). 


The ANOVA results of the Model 2 are also found in Table 
3. 


If the inverse ilr transformation is performed to the esti- 
mated coefficient vector of Model 2, the coefficient vector 
in S? is obtained, (LIT,INF,CRIT) = (0.28,0, 35,0, 37) 
which allows to conclude that the emphasis on certain activ- 
ities allows to obtain a better performance, still subtracting 
the effect of the context. 


Figure 5 shows the level curves of the estimated regression 
function in Model 2. Even when controlling for the context 
variables, teachers who emphasize the activities of critical 
reading versus literal reading obtain —on average— a better 
group performance in the reading test. Higher intensities of 
gray indicate larger group performance. 


It is important to highlight that Model 2 has high variabil- 
ity, which makes it possible to conjecture that other group, 
school or teacher variables that influence performance are 
not being accounted for in the model. 


5. DISCUSSION AND CONCLUSIONS 


This paper analyzes the relationship between the empha- 
sis placed by teachers on different classroom reading activ- 
ities (literal, inferential and critical reading), and the read- 
ing performance of their nine grade students in Uruguay. 
This makes it a novel contribution to educational policy in 
Uruguay as the topic has never been studied, and it is also 
a contribution to educational evaluation, as it applies the 
analysis of compositional data to this field of research. 


Findings show a relationship between reading test scores 
and the emphasis placed by teachers on different reading 
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INF 


Figure 5: Level curves set of the estimated regression func- 
tion in Model2. 


dimensions. Better test results are associated with teacher 
emphasis on critical reading activities. Nonetheless, this re- 
lationship is not direct as it is affected by other variables 
such as the socioeconomic and cultural level of the group 
of students and their reading habits. However, even when 
controlling for these variables, the evidence holds: greater 
emphasis placed on critical reading activities versus literal 
reading is associated with higher group results on the read- 
ing test. 


Moreover, findings also suggest the omission of certain vari- 
ables that can be relevant to explain reading performance by 
students. It could be the case of prior reading achievement. 
Although it could be argued that teachers place more em- 
phasis on critical reading activities when their students are 
better prepared for that (when they have higher prior read- 
ing achievement), evidence from primary school students 
from Uruguay shows that is not necessarily the case, see 
[16]. Nonetheless, this will be explored in future research. 


In Uruguay, the differences in academic achievement are 
strongly related to socieconomic factor, see [23], [2] and [3]. 
This study shows that having those factors controlled, the 
OTL offered to student variable is still relevant on achieve- 
ment data. The dimension of this finding has implications 
not only on the teaching profession but also on public edu- 
cational policies. 
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